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Summary 


•  The  STP97  data  set  is  suitable  for  use  in 
providing  current  norms  for  the  1 0^^,  1 and 
12th  grade  students 

-  Current  Population  Survey  (CPS)  and  STP  critical 
demographics  agree 

-  National  Assessment  of  Educational  Progress 
(NAEP)  and  STP  score  changes  from  1980  to 
1 997  agree 

•  The  STP  data  set  for  2-year  college  students 
is  problematic 

•  Options  for  developing  STP  norms  are 
outlined 


This  slide  summarizes  the  results  of  our  evaluation  of  the  Student  Testing 
Program  (STP)  Norming  Sample.  First,  STP97  is  a  suitable  data  set  to  use  to 
provide  current  norms  for  students  in  grades  10  through  12.  Second,  the  STP 
data  set  for  2- year  college  students  has  problems  and  might  be  improved  by 
reweighting.  Third,  we  present  four  options  for  developing  STP  norms. 
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Background 


•  The  STP  provides  a  form  of  ASVAB  to  high 
schools  and  postsecondary  schools  for  use 
in  career  exploration 

•  National  norms  are  provided  for  students  in 
-Grades  10, 11, 12 

-  Postsecondary  (2-year)  colleges 

•  Current  STP  norms  are  based  on  data 
from  PAY80  and  are  thought  to  be  dated 


The  Department  of  Defense  sponsors  the  STP,  which  provides  a  form  of  the 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  for  use  in  high  schools 
and  postsecondary  schools.  The  test  scores  are  used  for  career  exploration  in 
the  schools  and  may  also  be  used  to  enlist  in  the  armed  forces. 

National  norms  are  provided  for  students  in  grades  10,  11,  and  12  as  well  as 
for  postsecondary  (2-year)  colleges.  These  norms  enable  students  to  know  how 
their  scores  compare  with  a  national  sample  of  youth  in  their  particular  grades. 

Current  STP  norms  are  based  on  data  collected  in  1980  as  part  of  the  Profile  of 
American  Youth  (PAY80)  data  collection.  PAY80  was  a  national  data 
collection  sponsored  by  the  Department  of  Labor  (DOL)  and  the  Department 
of  Defense  (DOD).  These  norms  are  thought  to  be  dated. 

STP  norms  are  used  to  provide  career  counseling  information  to  students  in 
thousands  of  high  schools  each  year.  It  is  important  that  these  norms  be 
correct.  To  this  end,  the  Defense  Manpower  Data  Center  (DMDC)  asked  CNA 
to  evaluate  the  suitability  of  STP  data  for  use  in  providing  norms  for  10**' 
through  12**'  grade  students  and  2- year  college  students,  and  to  summarize  the 
options  for  developing  STP  norms. 
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Background  (continued) 


•  New  ASVAB  data  are  now  available 
from  tests  administered  as  part  of 
the  PAY97 

- STP97,  grades  10,  11,  and  1 2  in  fall  of 
1997 

-  ETP97,  ages  1 8-23  on  June  1 ,  1 997 


New  ASVAB  data  are  now  available  from  tests  administered  in  1997  as  part  of 
a  joint  DOL/DOD  effort  known  as  PAY97.  This  data  eollection  was  done  by 
the  National  Opinion  Research  Center  (NORC)  and  is  part  of  the  National 
Longitudinal  Survey  of  Youth  (NLSY97). 

We  will  examine  two  subsets  of  the  PAY97  data: 

•  STP97,  which  contains  ASVAB  scores  for  students  expected  to  be  in 
grades  10,  11,  and  12  during  the  fall  of  1997 

•  Enlistment  Testing  Program  97  (ETP97),  which  contains  scores  of  older 
youth  ages  18-23  on  June  1,  1997.  We  will  use  this  data  set,  although 
not  called  “STP97”  in  official  documentation  of  the  PAY97  data  set,'  to 
examine  data  for  youth  in  2- year  colleges. 


1.  Whitney  Moore,  Steven  Pedlow,  and  Kirk  Wolter.  Profile  of  American  Youth  1997  (PAY97) 
Technical  Sampling  Report,  NORC,  Aug  1999. 
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Issue 


•  Are  the  PAY97  data  of  sufficient 
quality  to  use  in  developing  new 
norms  for  the  STP? 


We  will  address  the  issue  of  whether  the  PAY97  data  are  of  sufficient  quality 
to  use  in  developing  new  norms  for  the  STP. 

In  previous  reports,^  we  have  raised  serious  questions  about  the  quality  of  the 
ETP97  data  intended  for  use  in  developing  norms  for  the  18-  to 
23-year-old  age  group. 


2.  William  H.  Sims  and  Catherine  M.  Hiatt.  Analysis  ofNLSY97  Test  Scores,  Jul  1999  (CNA 
Annotated  Briefing  99-66). 

William  H.  Sims  and  Catherine  M.  Hiatt.  Follow-on  Analysis  ofPAY97  Test  Scores,  Jul  2001 
(CNA  Annotated  Briefing  D0003839.A2). 
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Criterion 


•  The  data  are  sufficient  if  they  are 
representative  of  the  underlying  target 
population  with  respect  to 
-Age 
-Gender 

-  Race-ethnicity 

-  Respondent’s  education 

-  Mother’s  education 


In  a  recent  report, ^  we  show  that,  in  general,  norming  data  must  be 
representative  of  the  underlying  target  population  with  respect  to  the 
demographic  variables  of  age,  gender,  race-ethnicity,  respondent’s  education, 
and  mother’s  education. 

We  will  consider  the  STP97  data  sufficient  if  they  are  representative  of  the 
underlying  target  population  with  respect  to  these  five  demographic  variables. 


3.  William  H.  Sims  and  Catherine  M.  Hiatt.  On  the  Representativeness  of  Norming  Samples 
for  Aptitude  Tests,  Oct  2002  (CNA  Annotated  Briefing  D0007188.A1). 
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Approach 


•  Compare  the  NORC-weighted 
STP97  distributions  with  those 
expected  from  the  CPS 

•  Compare  score  changes  between 
1980  and  1997  using  NAEP  and 
STP 


Our  approach  has  two  main  thrusts. 

First,  we  will  compare  the  distributions  of  the  five  critical  demographic 
variables  using  population  weighted  (by  NORC)  STP97  data  with  distributions 
of  the  same  variables  from  the  Current  Population  Survey  (CPS)."^ 

In  addition,  we  will  compare  STP  score  changes  between  1980  and  1997  with 
those  seen  in  an  independent  assessment  of  ability  from  the  National 
Assessment  of  Educational  Progress  (NAEP).^ 


4.  Bureau  of  Labor  Statistics/Bureau  of  the  Census.  Current  Population  Survey  (Series). 

5.  National  Council  on  Educational  Statistics  (NCES),  NAEP  1999  Trends  in  Academic 
Progress,  2000. 
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STP  grades  10, 11,  and  12 


First,  we  will  address  the  STP  data  for  students  entering  grades  10,  11,  and  12 
in  the  fall  of  1997. 
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Selection  for  grades  10, 11,  and  12 


•  STP  sample  with  NORC  poststratification 
weights  by  gender  and  grade 

•  Selections 

-  Eligs  =‘s’ 

-Wt6s  >0 

-  Fgrade97=1 0,  11 ,  or  1 2 

-  4,652  cases 


We  selected  a  data  sample  that  had  been  poststratification  weighted  by  NORC 
on  gender  and  grade.  A  total  of  4,652  cases  were  found  with  the  proper 
eligibility  code,  positive  case  weights,  and  expected  grade  in  Ml  1997  of  10, 
11,  or  12. 
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Minor  issue 


•  Unlike  ETP,  the  STP  sample  was  not 
reconfigured  and  reweighted  to  include 
language  barrier  cases  and  exclude  outliers 

•  If  STP  were  reconfigured  and  reweighted  as 
was  ETP,  we  estimate  that  it  would  reduce 
the  mean  AFQT  by  0.33  percentile  point 

•  We  have  ignored  this  issue  in  the  following 
analysis 


Before  we  examine  the  data  in  detail,  we  must  dispense  with  a  minor  issue. 

Planners  of  PAY97  had  intended  that  all  persons  eligible  for  testing  be  tested 
regardless  of  their  ability  to  speak  or  read  English.  Late  in  the  data  collection, 
however,  we  discovered  that  test  administrators  had  not  administered  the  test 
to  about  250  persons  who  were  considered  not  to  have  a  functional  facility 
with  the  English  language.  These  persons  are  referred  to  as  “language  barrier” 
cases.  The  ETP  sample  has  been  reconfigured  with  an  option  to  use  special 
weights  and  imputed  data  for  these  cases. 

Unlike  the  ETP,  the  STP  sample  was  not  reconfigured  and  reweighted  to 
include  language  barrier  cases  and  to  exclude  outliers.  We  estimate  that,  if 
STP  were  reconfigured  and  reweighted  as  was  ETP,  the  mean  AFQT  would  be 
reduced  by  about  0.33  percentile  point.  This  is  a  very  small  amount,  and  we 
have  ignored  this  issue  in  the  following  analysis. 
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Comparison  of  STP  and  CPS 
demographics 


In  this  section,  we  compare  distribution  of  STP  and  CPS  data  with  regard  to 
the  five  critical  demographic  variables. 
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STP  vs.  CPS  by  grade 


Good  agreement! 

This  result  was  expected 
given  that  NORC  weighted 
grade  and  gender  to  CPS97 


Grade  (fall  1997) 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  grade  of  youth  in  the 
fall  of  1997.  Differenees  are  small  (about  0.2  pereentage  point)  and  the 
agreement  is  good.  This  result  was  expected  given  that  NORC  had  done  post¬ 
stratification  weighting  by  grade  and  gender. 
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STP  vs.  CPS  by  gender:  grades  10-12 


Good  agreement. 


This  result  was  expected  given 
that  NORC  weighted  gender 
and  grade  to  CPS  97 


Gender 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  gender  of  youth.  The 
differenees  between  STP  and  CPS  are  very  small  (less  than  0.2  percentage 
point),  and  the  agreement  is  good.  This  result  was  expected  given  that  NORC 
had  done  poststratification  weighting  by  grade  and  gender. 
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STP  vs.  CPS  by  race  /ethnicity: 
grades  10-12 


Good  agreement! 


Race/ethnicity 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  race/ethnicity  of 
youth.  The  agreement  is  good. 


14 


STP  vs.  CPS  by  age:  grades  10-12 


Poor  agreement! 

As  a  result  of  definition  of 
"fagestp"  age  variable  in  SCF 


Age 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  age  of  youth.  At  first 
glance,  the  agreement  is  poor!  On  closer  inspection,  however,  we  traced  the 
discrepancy  to  the  complex  and  unsuitable  definition  of  the  “fagestp”  variable 
in  the  sample  control  file  (SCF). 

We  recomputed  the  age  variable  in  the  STP  sample  to  be  age  as  of  October 
1997  (same  date  used  by  the  CPS)  and  show  the  results  on  the  following  slide. 
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STP  vs.  CPS  by  age  in  October  1997: 
grades  10-12 


Good  agreement! 

STP  age  variable  redefined 
as  of  October  1997 


'  STP_10-12Oct 
*  cpsjroT 


Age 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  age  of  youth  in  the  fall 
of  1997.  The  agreement  is  good. 
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STP  vs.  CPS  by  mother’s  education: 
grades  10-12 


Good  agreement! 


Mother's  education 


In  this  chart,  we  compare  STP  and  CPS  distributions  by  mother’s  educational 
level.  The  agreement  is  good. 

In  appendix  C,  we  evaluate  alternative  data  sets  for  estimating  mother’s 
edueational  level  from  CPS.  We  conelude  that  the  CPS97  data  (used  here)  are 
preferred. 
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STP  postsecondary  sample 


We  now  address  the  STP  postsecondary  sample. 
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Mean  AFQT  by  type  of  school 
attended:  PAY97  ETP  sample 


type  of  school  attending/last  attended 

Cases  weighted  by  WT6EOUT 


The  STP  refers  to  2- year  colleges  as  postsecondary  schools.  This  chart  shows 
the  mean  AFQT  score  by  type  of  school  attending  or  last  attended.  The  2- year- 
college  students  appear  to  be  a  unique  group  and  are  not  appropriate  for 
combining  with  other  educational  groups.  Their  AFQT  scores  fall  between 
those  of  persons  in  vocational  schools  and  those  in  4-year  colleges. 
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Selection  for  2-yr  college 
(aka  postsecondary  schools) 


•  ETP  sample  with  NORC  poststratification 
weights  by  age,  gender,  and  race/ethnicity 

•  Selections: 

-  Eligeout  =  ‘e’ 

-  WtSeout  >0 

-  1 ,202  cases 

-  Online  questionnaire,  question  #3 

•  What  type  of  school  are  you  now  attending,  or  did 
you  last  attend?  Option  7  (2-yr  college) 

•  Same  selections  as  in  STP80 


This  slide  describes  the  data  selections  we  made  to  get  the  2- year  college 
sample.  These  selections  are  the  same  as  those  used  in  STP80. 

There  is  inconsistency  in  nomenclature  that  may  lead  to  some  confusion. 
Department  of  Defense  literature®  describes  the  STP  as  consisting  of  persons  in 
grades  10,  11,  and  12  and  in  2-year  colleges  and  makes  available  norms  for 
each  group.  However,  NORC  NLSY97/PAY97  documentation’  considers 
persons  in  grades  10,  11,  and  12  to  be  in  the  STP  and  those  in  2-year  colleges 
to  be  in  the  ETP.  In  this  report,  we  adhere  to  the  DOD  definition.  Hence,  we 
will  be  using  the  PAY97/ETP  data  for  STP  persons  in  2- year  colleges. 


6.  Technical  Manual  for  the  ASVAB  18/1 9  Career  Exploration  Program,  U.S.  Department  of 
Defense,  1994. 

7.  Whitney  Moore,  Steven  Pedlow,  and  Kirk  Wolter.  Profile  of  American  Youth  1997 
(PAY97)  Technical  Sampling  Report,  NORC,  Aug  1999. 
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STP  vs.  CPS  by  gender:  2-yr  college 


Gender 


This  slide  compares  STP  data  by  gender  with  the  distributions  expected  from 
CPS97.  The  agreement  is  marginal.  The  STP  data  show  about  1  percentage 
point  too  many  males. 
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STP  vs.  CPS  by  race/ethnicity: 
2-yr  college 


Poor  agreement: 


Race/ethnicity 


This  slide  compares  STP  data  by  race/ethnicity  with  the  distributions  expected 
from  CPS97.  The  agreement  is  poor.  The  STP  data  show  about  5  percentage 
points  too  many  whites. 
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STP  vs.  CPS  by  age  in  October  1997: 
2-yr  college 


Poor  agreement, 
but  is  an  artifact  due  to 
the  broad  definition 
of  2-yr  college  group 


Age  (October  1997) 


This  slide  compares  STP  data  by  age  with  the  distributions  expected  from 
CPS97.  Superficially,  the  agreement  is  poor.  However,  this  is  an  artifact  due  to 
the  broad  definition  of  the  2-year  college  group. 

In  keeping  with  current  DOD/STP  practice  (and  the  precedent  set  with 
STP80),  we  defined  the  2- year  group  as  those  now  attending  or  who  have  last 
attended  a  2- year  college.  Obviously,  this  is  a  somewhat  older  group  than  the 
CPS  group — that  is,  those  attending  a  2- year  college  in  the  fall  of  1997.  We  do 
not  consider  the  difference  in  age  distributions  to  be  a  serious  problem. 
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STP  vs.  CPS  by  mother’s  education: 
2-yr  college 


Poor  agreement: 
missing  children  of  highly 
educated  mothers 


Mother's  education 


This  slide  compares  STP  data  by  mother’s  education  with  the  distributions 
expected  from  CPS97.  The  agreement  is  marginal.  The  STP  data  set  appears  to 
be  missing  children  of  highly  educated  mothers. 

There  is  some  disagreement  about  which  CPS  data  set  provides  the  best  target 
populations.  We  examine  this  issue  in  appendix  C  and  conclude  that  the 
various  alternatives  overestimate  highly  educated  mothers  by  about  0.8  to  2.2 
percentage  points.  We  find  that  the  CPS97  is  preferred  to  the  alternatives. 
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Illustrative  STP97  norm  table:  12**^ 
grade  combined  gender  norms  for  WK 


It  takes  a  higher  score  in  1997 
to  reach  the  same  cumulative 
percentile.  12^^  grade 
students  in  1997  scored  somewhat 
higher  than  those  in  1980. 


This  slide  illustrates  what  an  STP  norming  table  might  look  like.  The  slide 
shows  12^^  grade  combined  gender  norms  for  the  Word  Knowledge  (WK) 
subtest  from  ASVAB. 

The  solid  line  shows  the  current  STP  norms  (STP80).  We  developed  the 
dashed  line  from  STP97  data  described  in  this  analysis.  Near  the  middle  of  the 
distribution,  standard  scores  (x-  axis)  translate  into  lower  percentile  scores  in 
the  1997  sample  than  in  the  1980  sample.  This  means  that  the  1997  sample 
scored  somewhat  higher  on  the  WK  subtest  than  the  1980  sample  (ie.,  it  takes 
a  higher  standard  score  in  1997  to  reach  the  same  cumulative  percentile  of  12'*' 
grade  students). 
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Mean  scores  1980  vs.  1997: 
STP  12**^  grade  norming  samples 


Mean 

Difference 

Test/subtest 

1980 

1997 

(std.  dev.  units) 

AFQT  percentile  score 

46.98 

49.60 

.09 

General  Science  (GS) 

48.76 

49.38 

.06 

Arithmetic  Reasoning  (AR) 

49.18 

48.92 

-.03 

Word  Knowledge  (WK) 

48.33 

49.26 

.09 

Paragraph  Comprehension  (PC) 

49.48 

47.82 

-.17 

Numerical  Operations  (NO) 

49.31 

51.73 

.24 

Coding  Speed  (CS) 

48.54 

50.91 

.24 

Auto  &  Shop  Information  (AS) 

47.51 

43.92 

-.36 

Math  Knowledge  (MK) 

50.17 

53.05 

.29 

Mechanical  Comprehension  (MC) 

48.87 

46.80 

-.21 

Electrical  Information  (El) 

47.71 

44.83 

-.29 

Verbal  (VE) 

48.63 

48.75 

.01 

This  slide  compares  the  mean  scores  for  12*  grade  students  in  1980  and  1997. 
We  also  show  the  difference  in  the  mean  values  expressed  in  standard 
deviation  units. 

Mean  scores  on  some  subtests  went  up  between  1980  and  1997,  and  some 
went  down.  Note  the  large  increases  in  NO,  CS,  and  MK  and  the  large 
decreases  in  PC,  AS,  MC,  and  El. 
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Comparison  with  NAEP 


•  NAEP  scale  scores  (1980  and  1997) 
- 1 7-year-olds 

-  Math  and  reading 

•  Compared  with  PAY80  and  PAY97 

-  Students  entering  1 2^^  grade  (fall  1 997) 

-  Average  age  1 7.4  years 

-  Math  (AR  +  MK)  and  verbal  (VE) 


In  this  section,  we  compare  changes  in  ASVAB  scores  over  time  with  changes 
in  an  external  benchmark  test. 

We  will  use  the  scale  score  data  from  the  National  Assessment  of  Educational 
Progress*  as  an  external  benchmark.  The  data  cover  17-year-old  youth  tested  in 
the  spring  of  various  years  on  math  and  reading  skills.  The  math  and  verbal 
scale  scores  on  the  NAEP  have  been  shown  to  be  highly  correlated  to  ASVAB 
(Bloxom). 

We  will  compare  NAEP  scores  with  ASVAB  math  and  verbal  scores  of 
persons  entering  12*  grade  in  the  fall  of  1997.  These  persons  have  an  average 
age  of  17.4. 


8.  National  Council  on  Educational  Statistics,  NAEP  1999  Trends  in  Academic  Progress,  2000. 
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NAEP  math  and  reading  scores  for 

17-year-olds:  1970-1999 
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Year  (19XX) 

This  chart  shows  math  and  verbal  scale  scores  for  17-year-old  youth  from 
1970  through  1999.  The  ehart  also  shows  years  when  PAY  (ASVAB)  data 
were  collected.  In  most  cases,  the  years  of  NAEP  testing  did  not  correspond  to 
years  of  PAY  testing.  We  average  the  NAEP  data  from  years  that  bracket  the 
PAY  years. 
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NAEP  and  ASVAB  show  very  similar 
changes  from  1980  to  1997 


m - 

Change:  1980  to 
1997 

Category 

Test 

Age 

“1980” 

“1997” 

Points 

Std.dev. 

unjl^ 

Math 

NAEP  math 

17 

299.5 

307.7 

8.20 

(!i) 

ASVAB  AR 

17.4 

49.18 

48.92 

-.26 

-.03 

ASVAB  MK 

17.4 

50.17 

53.05 

2.88 

.29 

ASVAB  (MK+AR)/2 

17.4 

Verbal 

NAEP  reading 

17 

285.5 

287.7 

2.20 

ASVAB  VE 

17.4 

48.63 

48.75 

0.12 

(?) 

Source:  NAEP  1999  Trends  in  Academic  Progress,  NCES 
Inferred  NAEP  verbal  std.  dev.  =75.2,  math  std.  dev.  =  71.4 


This  chart  shows  mean  NAEP  and  PAY  (ASVAB)  scores  for  the  “1980”  and 
“1997”  testing  for  youth  of  comparable  ages. 

We  used  the  PAY97  STP  weights  for  this  analysis.  The  average  increase  in 
NAEP  math  scores  was  0.12  standard  deviation.  The  average  change  in 
ASVAB  math  scores  was  0.13  standard  deviation.  The  average  change  in 
NAEP  verbal  scores  was  0.03  standard  deviation,  and  that  for  ASVAB  verbal 
was  0.01.  All  ASVAB  scores  are  on  the  1980  score  scale. 

These  changes  in  scores  over  this  17-year  interval  are  very  consistent  between 
the  two  tests,  and  they  support  our  conclusion  that  the  STP  97  sample  is  a  good 
one. 
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STP  norming  options 


•  Norm  STP97  data  using  new  score  scale 
developed  from  ETP97 

-  Feasible  if  ETP97  data  support  a  reliable  score  scale 

•  Develop  new  STP97  score  scale  and  use  it  to 
norm  STP97  data 

-  Would  give  the  STP  its  own  new  score  scale  and  new  norms 

•  Norm  STP97  data  using  old  1 980  score  scale 

-  Would  give  STP  new  norms  based  on  performance  of  high 
school  students  in  1997 

•  Do  nothing 

-  Continue  to  use  existing  STP  norms  based  on  1980  students 


Based  on  the  foregoing  discussion,  it  seems  that  there  are  four  broad  options 
for  STP  norms: 

•  Norm  STP97  data  expressing  subtest  standard  scores  in  a  new  score 
scale  developed  from  ETP97.  This  is  feasible  only  if  the  ETP  data  are  of 
sufficient  quality  to  support  a  reliable  score  scale. 

•  Develop  a  new  STP  score  scale  and  express  subtest  standard  scores  in  it 
to  norm  STP97  data.  This  would  give  the  STP  its  own  score  scale  and 
new  norms. 

•  Norm  STP97  data  expressing  subtest  standard  scores  on  the  old  1980 
score  scale.  This  would  give  the  STP  new  norms  based  on  the 
performance  of  high  school  students  in  1997.  Of  course,  the 
intermediate  step  in  forming  the  norms  would  use  an  old  score  scale,  but 
that  should  not  present  any  technical  problems. 

•  Do  nothing.  DOD  could  continue  to  use  the  STP80  norms. 
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Summary 


•  The  STP97  data  set  is  suitable  for  use 
in  providing  current  norms  for  the 

and  grade  students 

-  CPS  and  STP  critical  demographics  agree 

-  NAEP  and  STP  score  changes  from  1 980 
to  1 997  agree 

•  The  STP  data  set  for  2-year  college 
students  is  problematic 

•  There  are  several  norming  options 


We  conclude  that  (a)  the  STP97  data  set  is  suitable  for  use  in  providing  current 
norms  for  the  lO'**,  1 1*,  and  12*  grade  students,  (b)  the  STP  data  set  for  2-year 
college  students  might  be  improved  by  reweighting,  and  (c)  four  norming 
options  exist. 
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Questions? 
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Appendix  A:  STP  data 


In  this  appendix,  we  tabulate  the  Student  Testing  Program  (STP)  data. 
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Disposition  of  STP97  eligibies 


CNTCCD 

Description 

WT6S  =  0 

WT6S  >  0 

Total 

Blank 

Not  in  sample 

207 

207 

30 

Parent  refused 

71 

71 

31 

Respondent  refused 

108 

108 

32 

No  show,  not  rescd. 

19 

19 

33 

Canceled  by  Sylvan 

225 

225 

35 

Other  no  show 

801 

1 

802 

36 

Unlocatable 

43 

43 

40 

Showed  up  for  test 

44 

4651 

4695 

Total 

1518 

4652 

6170 
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STP  data:  distribution  in  age 


STP  percentage  by  age  by  grade  in  fall  1997 

Age  (10/97) 

10th 

11th 

12th 

Total  10-12 

2-yr.  college 

12 

.00 

.00 

.10 

.00 

13 

.50 

.10 

.10 

.20 

14 

7.40 

.50 

.30 

2.90 

15 

58.50 

10.00 

.80 

24.10 

16 

27.80 

61.90 

8.10 

32.30 

.00 

17 

4.30 

23.10 

57.90 

27.90 

.00 

18 

.90 

3.20 

26.40 

10.00 

7.50 

19 

.30 

.40 

3.80 

1.40 

19.40 

20 

.10 

.30 

.90 

.50 

20.70 

21 

.10 

.20 

.70 

.30 

16.50 

22 

.00 

.10 

.50 

.20 

16.50 

23 

.10 

.10 

.40 

.20 

13.60 

24 

5.60 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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STP  data:  distribution  in  race/ethnicity 


STP  percentage  by  race/ethnicity  by  grade  in  fall  1997 

Race/ 

Ethnicity 

IQth 

11th 

12th 

Total  10-12 

2-yr.  college 

Non-Black  non- 
Hispanic 

68.40 

72.00 

71.00 

70.40 

69.40 

Black 

17.20 

14.80 

15.90 

16.00 

15.50 

Hispanic 

14.30 

13.30 

13.00 

13.60 

15.10 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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STP  data:  distribution  in  gender 


STP  percentage  by  gender  by  grade  in  fall  1997 

Gender 

IQth 

11th 

12th 

Total  10-12 

2-yr.  college 

Male 

51.40 

50.70 

50.40 

50.80 

47.10 

Female 

48.60 

49.30 

49.60 

49.20 

52.90 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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STP  data:  distribution  in  mother’s 
education 


STP  percentage  by  mother’s  education  by  grade  in  fall  1 997 

Mother’s 

education 

10“1 

11th 

12th 

Total  10-12 

2-yr.  college 

8 

4.6 

5.0 

4.5 

4.7 

4.2 

9 

3.1 

2.7 

1.6 

2.5 

1.4 

10 

3.8 

3.6 

4.0 

3.8 

2.2 

11 

5.2 

4.1 

4.3 

4.5 

3.2 

12 

34.7 

39.1 

37.9 

37.1 

47.0 

14 

27.2 

23.7 

25.3 

25.5 

23.4 

16 

21.4 

21.8 

22.4 

21.8 

18.6 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 
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STP  data:  distribution  in  grade 


Grade  in  fall  1997 

STP 

10 

35.2 

11 

32.1 

12 

32.7 

Total 

100.0 
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Appendix  B:  CPS  data 


In  this  appendix,  we  show  the  Current  Population  Survey  (CPS)  data. 


41 


CPS  data:  distribution  in  age 


CPS  percentage  by  age  by  grade  in  fall  1997 

Age  (10/97) 

10‘h 

11th 

12th 

Total  10-12 

2-yr.  college 

12 

.00 

.00 

.00 

.00 

13 

.10 

.20 

.10 

.10 

14 

4.60 

.30 

.20 

1.80 

15 

62.50 

6.50 

.60 

24.10 

16 

27.10 

60.70 

6.00 

31.00 

.30 

17 

4.50 

26.60 

63.60 

31.00 

3.00 

18 

.70 

4.40 

22.80 

9.20 

22.30 

19 

.10 

.70 

4.50 

1.70 

24.00 

20 

.10 

.30 

1.20 

.50 

21.40 

21 

.10 

.20 

.40 

.20 

10.30 

22 

.00 

.00 

.30 

.10 

11.00 

23 

.20 

.00 

.40 

.20 

7.70 

24 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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CPS  data:  distribution  in  race/ethnicity 


CPS  percentage  by  race/ethnicity  by  grade  in  fall  of  1997 

Race/ 

Ethnicity 

IQth 

11th 

12th 

Total  10-12 

2-yr.  college 

Non-Black  non- 
Hispanic 

70.40 

70.40 

72.10 

71.00 

74.90 

Black 

16.60 

16.10 

15.90 

16.20 

11.40 

Hispanic 

13.00 

13.50 

12.00 

12.80 

13.70 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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CPS  data:  distribution  in  gender 


CPS  percentage  by  gender  by  grade  in  fall  of  1 997 

Gender 

10th 

11th 

12th 

Total  10-12 

2-yr.  college 

Male 

51.40 

50.70 

50.50 

50.90 

45.90 

Female 

48.60 

49.30 

49.50 

49.10 

54.10 

Total 

100.00 

100.00 

100.00 

100.00 

100.00 
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CPS  data:  distribution  in  mother’s 
education 


CPS  percentage  by  mother’s  education  by  grade  in  fall  1 997 

Mother’s 

education 

10“1 

11th 

12th 

Total  10-12 

2-yr.  college 

8 

5.5 

5.5 

4.4 

5.1 

5.3 

9 

2.4 

1.4 

1.2 

1.7 

1.8 

10 

3.4 

2.2 

2.3 

2.7 

2.1 

11 

2.9 

2.2 

2.2 

2.4 

1.1 

12 

38.9 

37.9 

38.5 

38.4 

37.6 

14 

27.5 

28.8 

28.5 

28.3 

33.0 

16 

19.3 

22.0 

23.0 

21.4 

19.0 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 

45 


CPS  data:  distribution  in  grade 


Grade  in  fall  1997 

CPS 

10 

35.0 

11 

32.2 

12 

32.8 

Total 

100.0 
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Appendix  C:  Evaluation  of  data  on  mother’s 
education  in  the  target  population 


This  appendix  describes  our  evaluation  of  available  data  on  mother’s 
education  in  the  target  population. 
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Possible  target  data  sources  for 
mother’s  education 


•  CPS97 

-  The  standard  population  survey 

-  Misses  mothers  whose  children  are  no  longer 
living  in  the  household 

•  CPS  1995  Marriage  and  Fertility 

-  Ties  the  mother  to  five  of  her  children  by  age  of 
child  regardless  of  whether  the  child  is  in  the 
household 

-  Latest  data  are  from  1995  and  introduce  error  if 
the  percentage  of  highly  educated  mothers  is 
growing  rapidly  over  time 


The  CPS97  is  the  standard  population  survey.  It  misses  mothers  whose 
ehildren  are  no  longer  living  in  the  household. 

The  1995  CPS  Marriage  and  Fertility  Supplement  data  set  ties  the  mother  to  up 
to  five  of  her  children  regardless  of  whether  the  child  is  still  in  the  household. 

It  introduces  error  if  the  percentage  of  highly  educated  mothers  is  growing 
rapidly  over  time. 

Both  data  sets  appear  to  be  somewhat  imperfect  for  our  purposes.  We  will 
attempt  to  estimate  the  errors  in  distributions  of  mother’s  education  for  the 
target  population  made  using  each  data  set. 
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Percentage  of  youth  missing  mother’s 


Youth  group 

Age  of  youth 

Age  18-23 

2-  or  4-yr  college 

Grades  10-12 

13 

0.0 

14 

2.3 

15 

0.0 

4.3 

16 

0.0 

6.0 

17 

15.5 

7.5 

18 

21.9 

15.2 

18.9 

19 

32.8 

18.5 

39.0 

20 

41.5 

29.2 

59.8 

21 

49.5 

36.0 

44.0 

22 

61.0 

51.1 

100.0 

23 

69.4 

56.7 

100.0 

Total 

/^5.4^ 

/^0.9^ 

The  development  of  target  distributions  in  mother’s  education  from  CPS97 
requires  the  construction  of  a  “household  roster”  in  the  data  for  each 
household.  The  youth  in  the  household  are  then  assigned  the  educational  level 
of  the  mother  in  the  household.  Unfortunately,  many  older  children  leave  the 
household  before  the  age  of  23  and  are  invisible  to  this  procedure.  In  this  slide, 
we  show  the  percentage  of  youth  who  do  not  have  an  identifiable  mother  in  the 
data  set  by  age  of  youth.  Mother’s  education  computed  from  CPS97  will  be 
missing  for  these  youth. 

The  percentage  of  missing  data  increases  with  age  of  the  youth.  This  is  as 
expected  because,  the  older  the  youth,  the  more  likely  it  is  that  they  have  left 
home  and  set  up  a  separate  household.  We  see  that  45.4  percent  of  the  18-  to 
23-year-old  group  is  missing  mother’s  education  information.  The  percentage 
missing  drops  to  30.9  percent  for  those  in  college.  These  large  losses  may  be 
problematic. 

Only  8.4  percent  are  missing  for  those  mothers  whose  children  are  in  grades  10 
through  12.  This  result  suggests  that  the  data  from  CPS  97  are  probably 
suitable  for  use  in  developing  target  distributions  for  the  STP  sample. 

The  next  question  is.  Do  the  mothers  that  we  find  have  a  distribution  in 
education  that  is  radically  different  from  that  of  the  total  population? 


49 


Educational  distribution  of  mothers 
we  would  miss  if  we  used  CPS97 


1 995  CPS  Marriage  and  Fertility  file 
(Mothers  with  children  ages  18-23) 

Mother’s  education 

All 

Mothers  we  would 
find  in  CPS97 

Mothers  we  would 
miss^  in  CPS97 

Less  than  high  school 

6.4 

6.3 

7.0 

Some  high  school 

8.7 

8.3 

10.7 

High  school  graduate 

41.6 

41.3 

42.7 

Some  college 

26.6 

26.3 

27.5 

College  graduate 

16.7 

17.8 

12.1 

Total 

100.0 

100.0 

100.0 

1.  Mothers  with  children  18-23  not  living  in  household.  As  a  result,  our  target  distribution  of 
mother's  education  from  CPS  97  would  have  about  0.8%  too  many  highly  educated  mothers. 


We  use  the  1995  CPS  Marriage  and  Fertility  (M&F)  data  set  to  see  if  the 
mothers  we  are  missing  by  using  the  CPS97  data  are  different  from  those  we 
are  not  missing. 

We  categorize  the  mothers  we  would  miss  as  those  who  have  one  or  more 
children  age  18-23  not  living  in  the  mother’s  household.  Those  mothers  whose 
children  live  in  her  household  will  be  categorized  as  “found.” 

The  difference  between  what  we  would  find  using  CPS97  (column  labeled 
“mothers  we  would  find  in  CPS97”)  and  the  correct  answer  (column  labeled 
“All”)  overestimates  the  number  of  highly  educated  mothers  by  about  0.8 
percentage  point  (26.3  -i-  17.8  -  26.6  -16.7  =  0.8).  This  is  a  rather  small  error. 
Even  though  CPS97  misses  a  large  percentage  of  highly  educated  mothers,  the 
educational  distribution  of  those  found  is  very  close  to  that  of  the  total 
population. 
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Time  trend  in  mother’s  education  from 
CPS  Marriage  &  Fertility  files 


Mothers  with  children  of  indicated  age 

18-23 

16-21  in  1995 

Mother’s  education 

1985 

1990 

1995 

1995 

Less  than  high  school 

9.0 

7.9 

6.7 

6.2 

Some  high  school 

15.1 

11.4 

9.2 

8.8 

High  school  graduate 

48.3 

47.3 

41.4 

41.0 

Some  college 

15.8 

18.3 

26.6 

26.7 

College  graduate 

11.7 

15.1 

16.2 

17.3 

Total 

100.0 

100.0 

100.0 

100.0 

We  are  somewhat  concerned  that  using  the  1995  CPS  M&F  files  may  miss 
some  growth  in  the  educational  level  of  mothers  between  1995  and  1997.  To 
examine  this  possibility,  we  calculated  distributions  in  mother’s  education 
using  CPS  M&F  files  from  1985,  1990,  and  1995.  We  see  that  the  fraction  of 
highly  educated  (some  college  or  college  graduate)  mothers  has  been  rising 
steadily. 

In  an  attempt  to  capture  part  of  that  rise,  we  recalculated  the  mother’s 
education  distribution  for  mothers  whose  children  were  ages  16  to  21  in  1995. 
These  children  would  have  aged  to  18  to  23  in  1997  when  the  NLSY97  data 
were  collected.  We  observe  that  the  percentage  of  highly  educated  mothers 
increases  by  about  1  percent  with  this  2- year  time  shift.  This  adjustment  only 
goes  part  way  in  correcting  for  the  use  of  1995  data  versus  1997  data. 

On  the  next  slide,  we  examine  the  rise  in  the  percentage  of  highly  educated 
mothers  in  more  detail. 
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Changes  in  percentage  of  highly  educated^  mothers 
with  children  age  18-23:  CPS  M&F  files 


Period 

%  Change / 
period 

%  Change / 
year 

%  Change / 

2  years 

1985  to  1990 

5.7 

1.1 

2.2 

1990  to  1995 

9.4 

1.9 

3.8 

1 995  with  youth  1 8-23 
to 

1 995  with  youth  1 6-21 

1.2 

0.6 

1.2 

1.  College  graduates  or  some  college 


This  slide  focuses  on  the  change  in  percentage  of  highly  educated  mothers 
over  time.  We  define  highly  educated  mothers  as  college  graduates  or  those 
with  some  college.  The  data  are  derived  from  the  previous  slide. 

We  see  that  the  percentage  of  highly  educated  mothers  has  risen  between  1  and 
2  percentage  points  per  year  from  1985  through  1995.  Hence,  it  seems 
plausible  that  it  may  have  risen  2  to  4  percentage  points  during  the  2  years 
between  1995  (latest  year  of  good  M&F  files)  and  1997  (year  of  NLSY97). 
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Estimated  overestimation  in  percentage  of  highly 
educated  mothers  with  children  age  18-23  in  1997 


Data  source 

Overestimation  Error 
(percentage  points) 

1995  CPS  M&F  (youth  age 
18-23  in  1995) 

2.2  to  3.8 

1995  CPS  M&F  (youth  age 
16-21  in  1995) 

1.0  to  2.6 

1997  CPS 
(youth  age  1 8-23) 

0.8 

This  slide  summarizes  the  estimated  overestimation  error  that  is  likely  made  in 
the  target  populations  of  highly  educated  mothers  with  children  age  18-23  in 
1997  from  various  data  sources. 

The  first  line,  taken  from  the  previous  slide,  shows  that  the  error  likely  ranges 
between  2.2  and  3.8  percentage  points.  If  we  select  children  age  16-21  (who 
will  be  age  to  18-23  by  1997),  the  estimate  error  is  reduced  by  1.2  percentage 
points  to  a  range  of  1.0  to  2.6  percentage  points. 

The  estimated  error  in  using  CPS97  directly  is  taken  from  slide  49  and  is  0.8 
percentage  point.  These  errors  are  rather  small  and  are  in  the  same  direction 
(overestimation  of  the  percentage  of  highly  educated  mothers).  We  favor  the 
estimate  from  the  CPS97  because  the  estimated  error  is  slightly  smaller,  and 
the  data  come  from  the  same  standard  database  that  we  use  for  other 
demographic  variables. 
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CPS-  and  ETP-based  estimates  of 
mother’s  education 


Percent  of  mothers  by  mothers  education 

Mother’s  education 

CPS  M&F  1995 

Mothers  with  children 
age  16-21 

CPS  1997 

Mothers  with  children 
age  1 8-23 

Less  than  high  school 

6.2 

7.5 

Some  high  school 

8.8 

7.7 

High  school  grad 

41.0 

38.6 

Some  college 

26.7  1 

44  f) 

25.4 

.  46.2 

College  grad 

17.3 

20.8  J 

Total 

100.0 

100.0 

In  this  slide,  we  compare  target  population  distributions  for  mothers  of 
children  age  18-23  in  1997  from  two  sources.  One  source  is  the  1995  CPS 
Marriage  and  Fertility  file,  which  we  have  modified  slightly  to  select  children 
age  16-21  in  1995  so  that  they  will  be  age  18-23  in  1997.  The  other  is  the 
standard  CPS97  file. 

The  two  results  are  actually  quite  similar.  If  we  consider  “highly  educated” 
mothers  to  be  either  college  graduates  or  those  with  some  colle^,  the  results 
differ  by  only  about  2  percentage  points.  These  differences  are  within  the 
estimated  range  of  errors  in  slide  51. 
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Limitations  on  CPS  M&F  data  fiies 


•  Useful  in  telling  us  what  the  distribution  of 
mother’s  education  should  look  like  in  NLSY97 

•  Not  so  useful  in  fixing  any  problem  discovered 
in  NLSY  demographics 

-  Will  only  directly  support  weights  by  gender,  age, 
and  mother’s  education 

-  Will  not  directly  support  weights  by  race  or  youth 
education 

•  May  possibly  be  combined  with  regular  CPS 
files  using  conditional  probabilities 


It  appears  that  the  1995  CPS  M&F  data  are  useful  to  tell  us  if  the  NLSY97  is 
representative  of  the  underlying  population  with  respeet  to  mother’s  education. 

However,  it  does  not  appear  to  be  particularly  useful  in  fixing  a  problem  with 
this  variable  should  one  be  found.  This  is  because  the  structure  of  the  file  does 
not  permit  us  to  develop  a  self-contained  multidimensional  matrix  of  what  the 
population  should  look  like  in  terms  of  the  five  critical  demographic  variables 
of  age,  race,  gender,  youth  education,  and  mother’s  education.  It  appears  to  be 
capable  of  supporting  weighting  corrections  based  on  age,  gender,  and 
mother’s  education  but  not  on  race  or  youth  education. 

It  may  be  possible  to  circumvent  the  above  limitation  by  combining  data  from 
the  1997  CPS  file  and  the  1995  CPS  M&F  file  using  conditional  probabilities. 
Such  an  approach  would  seem  likely  to  introduce  additional  sampling  error. 
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Findings  with  respect  to 
data  on  mother’s  education 


CPS  data  on  mother’s  education  are 
unsatisfactory,  in  general 

-  No  one  CPS  data  set  has  good  information  on  all  5 
critical  demographics 

•  CPS97  data  are  good  for  age,  gender,  race,  and  youth 
education,  but  are  missing  mother’s  education  for  many  youth 

•  CPS95  Marriage  &  Fertility  file  is  good  for  youth  age,  gender, 
and  mother’s  education,  but  has  no  information  on  race  or 
youth  education  and  is  2  years  too  early  in  time 

However,  the  errors  made  in  estimating  mother’s 
education  level  by  either  method  are  not  large  and 
range  from  0.8  to  2.2  percentage  points. 

•  We  favor  the  use  of  CPS97  for  estimating  mothers  education 


In  general,  existing  CPS  data  on  mother’s  education  are  unsatisfactory.  No 
single  CPS  data  set  has  good  information  on  all  five  critical  demographics. 
The  CPS97  data  are  good  for  age,  gender,  race,  and  youth  educatbn  but  are 
missing  mother’s  education  for  youth  who  are  not  living  in  the  mother’s 
household.  The  CPS95  Marriage  and  Fertility  file  is  good  for  youth  age, 
gender,  and  mother’s  education,  but  provides  no  information  of  the  race  or 
education  of  the  mother’s  children. 

However,  the  good  news  is  that  the  errors  made  in  estimating  mother’s 
education  level  by  either  method  are  not  large  and  range  from  0.8  to  2.2 
percentage  points.  We  favor  the  use  of  CPS97  for  estimating  mother’s 
education  because  it  seems  to  have  the  smallest  error. 
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